Topology preserving intensity binning on reduced resolution grid of adaptive weighted cells

ABSTRACT

A camera system including: an image sensor that is controlled by one or more camera parameters; and a pre-processing circuit adapted to associate weights respectively with cells of a grid, wherein the weights differ from the one or more camera parameters, wherein the cells respectively include a plurality of contiguous picture elements of the image sensor.

CROSS REFERENCE

This application is a continuation of US patent application filing dateFeb. 17, 2014 Ser. No. 14/181,861 which is incorporated herein byreference.

BACKGROUND Technical Field

The present invention relates to pre-processing in a camera.

Description of Related Art

Image histogram calculations are present on many modern digital cameras.Photographers may use the image histogram as an aid to show thedistribution of tones captured, and whether image detail has been lost.In a histogram, the horizontal axis of the graph may represent tonalvariations, red/green/blue and/or gray scale. The vertical axis mayrepresent the number of pixels in each tone.

During the last few years camera based driver assistance systems (DAS)have been entering the market; including lane departure warning (LDW),automatic high-beam control (AHC), traffic sign recognition (TSR)forward collision warning (FCW) and pedestrian detection.

BRIEF SUMMARY

Various camera systems are provided for herein including an image sensorand a pre-processing circuit adapted to associate weights respectivelywith cells of a grid. The cells respectively may include multiplecontiguous picture elements of the image sensor. The pre-processingcircuit may be configured to adaptively change the weights responsive tochanges in the scene being imaged by the image sensor. The number of thecells per distance on the surface of the image sensor may besubstantially less than the resolution of the image sensor. The cellswhich share the same weights may specify a region of interest on theimage sensor. The weights associated with the cells of the region ofinterest may be non-zero and the weights associated with cells of thegrid outside the region of interest may be zero. The camera is mountableon a vehicle, and the region of interest may include an image of a roadin the environment of the vehicle.

An image processor may be connected to the image sensor. Thepre-processing circuit may calculate a pre-processing result over theregion of interest. The image processor may be configured to receive thepre-processing result from the pre-processing circuit. The imageprocessor responsive to the pre-processing result may reset a cameraparameter such as gain, exposure time and/or aperture.

The pre-processing circuit may perform a weighted histogram by summinginstances of a key over the cells of the grid. The key may include theweights as a multiplicative factor. The key may be a function of animage parameter such as image intensity, color intensity of at least onecolor, gradient of image intensity and/or gradient of color intensity.

The image processor may estimate ego-motion of the image sensor. Theimage processor, responsive to the estimated ego-motion of the camera,may reset one or more weights of one or more of the cells. The imageprocessor may estimate image motion of an object in the field of view ofthe image sensor and responsive to the estimated image motion reset oneor more of the weights of one or more of the cells.

Various computerized methods are provided herein for pre-processingimage data of an image frame in an image sensor. Weights are associatedrespectively with cells of a grid. The cells respectively may includemultiple contiguous picture elements of the image sensor. The weightsmay be adaptively changed responsive to changes in the scene beingimaged by the image sensor. Multiple picture elements of the imagesensor may be grouped contiguously into the cells of the grid with anumber of cells per distance on the surface of the image sensorsubstantially less than the resolution of the image sensor. The imagesensor may be mounted on a vehicle as a part of a driver assistancesystem. A region of interest may be determined to include cells ofnon-zero weight which image at least a portion of a road in theenvironment of the vehicle when traveling. Responsive to the weights, apre-preprocessing result may be calculated. A camera parameter may bereset responsive to the pre-processing result. A weighted histogram maybe performed by summing instances of a key over the cells of the grid.The key may include the weights as a multiplicative factor. The key maybe a function of at least one image parameter: image intensity, colorintensity of at least one color, gradient of image intensity and/orgradient of color intensity. Ego-motion of the image sensor may beestimated and responsive to the estimated ego-motion one or more of theweights of one or more of the cells may be reset. Image motion of anobject being tracked in the field of view of the image sensor may beestimated and responsive to the estimated image motion, one or more ofthe weights of one or more of the cells may be reset.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIGS. 1 and 2 illustrate a system including a camera or image sensormountable in a vehicle, according to an aspect of the present invention.

FIG. 3 shows an image frame from a road scene according to a feature ofthe present invention.

FIG. 4a illustrates region of interest superimposed onto an image framerespectively, according to a feature of the present invention.

FIG. 4b illustrates the region of interest re-projected onto the grid ofthe image frame due to ego-motion of the camera or image motion of anobject being tracked, according to a feature of the present invention.

FIG. 5, shows a simplified block diagram of pre-processing circuitryaccording to aspects of the present invention.

FIG. 6 shows a flow diagram of a process according to aspects of thepresent invention.

DETAILED DESCRIPTION

Reference will now be made in detail to features of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The features are described below to explain the presentinvention by referring to the figures.

Before explaining features of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of design and the arrangement of the components to set forth inthe following description or illustrated in the drawings. The inventionis capable of other features or of being practiced or carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein is for the purpose of description and shouldnot be regarded as limiting.

By way of introduction, various embodiments of the present invention aredirected to defining a region of interest in one or more image frames.In general, the region of interest may be selected by including an imageportion having the high variation in dynamic range over time from imageframe to image frame. Portions of the image having little variation indynamic range such as sky or landscape may be excluded from the regionof interest. The region of interest in image frames of camera baseddriver assistance systems may include the image of the road shown belowthe horizon, where images of other vehicles, traffic signs pedestriansand road obstructions are found. The position, shape and size of theregion of interest may be specific for one or more monitor and/orcontrol algorithms. The image may be pre-processed in the region ofinterest such as by computing a histogram which may be used by themonitor/control algorithm. A feature of the present invention includes asimple and efficient adjustment of the position of the region ofinterest over the image plane by keeping the topology of the region ofinterest similar to a canonical topology selected as appropriate for themonitor/control algorithm. The region of interest may be specified usingmultiplicative weighting factors or weights over a spatially constantgrid of reduced resolution in the image area. The weights provideflexibility of preserving the predefined topology of region of interestas well as a simple ability to adjust the position of the region ofinterest by varying the values of the weights typically over arelatively small number of cells at the periphery of the grid. The finedetails of the periphery of the region may be smoothed to the level ofthe reduced resolution by scaling the weights of the peripheral cells tobe proportional to the area of the cells within the region of interest.

Embodiments of the present invention may be useful to provide flexibleand efficient method of controlling the location of the region ofinterest over the image surface as a result of ego-motion of the cameraor image motion of an object of interest being tracked. In driverassistance systems, improved obstruction detection in the roadenvironment may be achieved when the obstruction and road have minimalimage contrast and/or in images which include the sun above the horizonoutside the region of interest.

Thus there is a need for and it would be advantageous to have a camerain which the region of interest of the image may be specified by weightsover a low resolution grid of cells in the image frames.

Referring now to the drawings, reference is now made to FIGS. 1 and 2which illustrate a system 16 including a camera or image sensor 12mountable in a vehicle 18, according to an aspect of the presentinvention. Image sensor 12, images a field of view in the forwarddirection. Image sensor 12 may be monochrome or black-white, i.e.without color separation or image sensor 12 may be color sensitive.Image frames 15 from camera 12 are captured and processed by imageprocessor 35. Camera 12 includes a pre-processing circuit 19 whichpreprocesses image frames 15 and may provide a monitor signal or mayreceive a control signal over a monitor/control bus 27 to or from imageprocessor 35 as a result of the pre-preprocessing. The monitor signalmay include histogram information being provided to image processor 35.

Image processor 35 may be used to process image frames 15 simultaneouslyand/or in parallel to serve a number of driver assistancesystems/applications. Image processor 35 may be used to process imageframes 15 to detect and recognize an image or portions of the image inthe forward field of view of camera 12. The driver assistance systemsmay be implemented using specific hardware circuitry (not shown) with onboard software and/or software monitor/control algorithms in storage 13.By way of example in FIG. 2, image frames 15 are used to servepedestrian detection 20, traffic sign recognition (TSR) 21 and forwardcollision warning (FCW) 22 according to features of the presentinvention.

In some cases, image frames 15 are partitioned between different driverassistance applications and in other cases the image frames 15 may beshared between the different driver assistance applications.

Reference is now made to FIG. 3 which shows an image frame 15 from aroad scene according to a feature of the present invention. Image frame15 in FIG. 3 shows a road scene as captured by image sensor 12. A regionof interest 30 is shown as a trapezoid and indicates an image of road infront of vehicle 18 as a region of image frame 15 which is of particularimportance for driver assistance functions such as pedestrian detection20, traffic sign recognition (TSR) 21, forward collision warning (FCW)22, detection of lane markers or obstructions in the road surface likemanhole covers. For other driver assistance applications or for imageprocessing in a field other than driver assistance, a region of interest30 of different size, shape, position and/or orientation in image spacemay be selected a priori.

Reference is now made to FIG. 4a which illustrates region of interest 30superimposed onto image frame 15 a respectively, according to a featureof the present invention. Picture elements in an image frame 15 acaptured by image sensor 12 are shown as parametrized by a coarse grid41, e.g. 50 by 50 pixels. Coarse grid 41 may be used to specify regionof interest 30. In order to specify region of interest 30, 8 cellslabeled individually with letters A-H are entirely within region ofinterest 30. The perimeter of region of interest, e.g. trapezoid 30crosses 16 cells, labeled with letter ‘p’.

Reference is now made also to FIG. 5, which is a simplified blockdiagram of preprocessing circuitry 19 and to FIG. 6 which is a flowdiagram 501 of a process according to aspects of the present invention.In FIG. 5, grouping circuitry 40 functions to group pixels (step 503)into cells of a low resolution grid, for instance grid 41 shown in FIG.4a . By selecting cells of grid 41, a region of arbitrary shape may beprovided (step 505). The region of arbitrary shape may be contiguous asin region of interest 30 as shown in FIGS. 3 and 4 a, or non-contiguous.Weighting circuitry 42, provides (step 507) a weight e.g. one, to allthe pixels of the selected cells, for instance cells A-H in region ofinterest 30 of FIG. 4a . For cells p which lie on the perimeter ofregion of interest 30, weighting circuitry 42 may assign a weightbetween zero and one which is proportional to the number of pixelswithin region of interest 30, or proportional to the cell area withinregion of interest 30 divided by the total cell area. Outside the regionof interest, the cells of grid 41 are normally weighted with smallerweights such as zeros. Histogram circuitry 44 may pre-process (step 509)image frames 15 responsive to the weights of the cells. Examples ofpre-processing may include calculating histogram information, averageand/or median of grey scale intensities or average color inRed/Green/Blue intensity values. The image information may be output(step 513) to image to processor 35 over monitor signal line 27. Acamera control parameter, e.g. gain, exposure time, for controllingcamera 12 may be reset (step 515) via control line 27 responsive to theinformation received in step 513.

In step 511, ego-motion of vehicle 18 may be detected and/or an image ofan object of interest may be tracked by image processor 35. Based on thedetected ego-motion and/or tracked image of an object of interest,region of interest 30 may be adjusted by signaling of control line 27.On receiving the signal, pre-processing circuit re-adjusts weights (step507) of the cells of grid 41 in order to maintain the tracked imagewithin region of interest 30 independent of the ego-motion of camera 12.For instance, the horizon image may be detected (step 511) which variesin image height dependent on the pitch angle of vehicle 18. Region ofinterest 30 may be maintained within 50 pixels for instance at the samevertical image distance relative to the imaged horizon independent ofthe changing pitch angle of vehicle 18, by adjusting weights (step 507)according cells of grid 41.

Reference is now also made to FIG. 4b which illustrates an example ofregion of interest 30 re-projected onto grid 41 of image frame 15 bresponsive to detected ego-motion of camera 12 and/or image motion of anobject being tracked, according to a feature of the present invention.Image frame 15 b of FIG. 4b illustrates for instance a case that thepitch angle of vehicle 18 is positive (upward) relative to the pitchangle of vehicle 18 when image frame 15 a of FIG. 4a is captured. Inorder to maintain tracking and/or region of interest 30 on the road atthe same distance always from vehicle 18, weights of cells of grid 41are re-adjusted (step 507). In the specific example of image frame 15 bof FIG. 4b , weights of cells labelled A,B,C,D,E,F,G,H remain the sameas the weights assigned to cells A-H in image frame 15 a of FIG. 4a .Region of interest 30 is updated and relocated by adjusting peripheralcells labelled I,J,K,L and additional cells p partially within region ofinterest 30. The use of low resolution grid 41 thus allows for acalculationally efficient mechanism for maintaining region of interest30 in the desired portion of images frames 15 since normally only asmall number of peripheral cells need to have respective weights updatedin step 507.

Although embodiments of the present invention are presented in thecontext of driver assistance applications, embodiments of the presentinvention may be equally applicable in other real time signal processingapplications and/or digital processing applications, such ascommunications, machine vision, audio and/or speech processing asexamples.

The term “resolution” as used herein refers to a number of pictureelements per distance for instance along a horizontal and/or verticalline on the surface of the image sensor.

The term “weight” as used herein refers to a multiplicative scale factorfor performing image processing over images. A weight of zero generallymeans that the pixel or group of pixels is not included in theprocessing. The higher the non-zero weight the more the pixel or groupof pixels is considered in the processing.

The term “contiguous” as used herein is defined herein by selecting twopoints of a geometric region and if the line segment connecting the twopoints wholly lies in the geometric region, then the geometric region iscontiguous.

The term “region of interest” as used herein refers to a region in imagespace the shape and dimensions of which remain essentially unchangedover multiple image frames. The term “region of interest” as used hereinis NOT an image of an object being tracked.

The term “ego-motion” as used herein refers to at least the angularre-orientation of the image sensor over angles” pitch, yaw and roll.

The term “key” as used herein refers to a parameter of interest selectedfor pre-processing an image. The “key” is used to represent a horizontalaxis and instances of particular values of the key are summed togenerate a histogram.

The indefinite articles “a”, “an” is used herein, such as “an image” hasthe meaning of “one or more” that is “one or more images”.

Although selected features of the present invention have been shown anddescribed, it is to be understood the present invention is not limitedto the described features. Instead, it is to be appreciated that changesmay be made to these features without departing from the principles toand spirit of the invention, the scope of which is defined by the claimsand the equivalents thereof.

I claim:
 1. A camera system including: an image sensor that iscontrolled by one or more camera parameters; a pre-processing circuitadapted to associate weights respectively with cells of a grid byassigning zero weights to cells of the grid located outside a region ofinterest, and assigning non-zero weights to cells of the grid that areincluded in the region of interest and to cells of the grid that areonly partially included in the region of interest, and to generate apre-processing result only for cells associated with non-zero weights;and an image processor operatively connectable to the image sensorconfigured to receive the pre-processing result from the pre-processingcircuit and modify a camera parameter of the one or more cameraparameters based on the pre-processing result; wherein the weightsdiffer from the one or more camera parameters, and wherein the cellsrespectively include a plurality of contiguous picture elements of theimage sensor.
 2. The camera system of claim 1, wherein the number of thecells per distance on the surface of the image sensor is substantiallyless than the resolution of the image sensor.
 3. The camera system ofclaim 1, wherein the pre-processing circuit is adapted to assign acertain non-zero weight for a certain cell that is only partiallyincluded in the region of interest based at least in part on an area ofthe cell that is within the region of interest.
 4. The camera system ofclaim 1, wherein the pre-processing circuit is adapted to assign acertain non-zero weight for a certain cell that is only partiallyincluded in the region of interest that is proportional to an area ofthe cell that is within the region of interest.
 5. The camera system ofclaim 1, wherein the image sensor is mountable on a vehicle, and whereinsaid region of interest includes an image of a road in the environmentof the vehicle.
 6. The camera system of claim 1, wherein thepre-processing circuit calculates a pre-processing result over theregion of interest, the camera system further comprising: an imageprocessor operatively connectable to the image sensor, wherein the imageprocessor is configured to receive the pre-processing result from thepre-processing circuit, wherein the image processor responsive to thepre-processing result resets the camera parameter of the one or morecamera parameters.
 7. The camera system of claim 6, wherein the cameraparameter is selected from the group consisting of: gain, exposure timeand aperture.
 8. The camera system of claim 6, wherein said cameraparameter is an aperture.
 9. The camera system of claim 1, wherein thepre-processing circuit is adapted to perform a weighted histogram. 10.The camera system of claim 1, wherein the pre-processing circuit isadapted to perform a weighted histogram by summing instances of a keyover the cells of the grid, wherein the key includes the weights as amultiplicative factor, and wherein the key is a function of at least oneimage parameter selected from the group consisting of: image intensity,color intensity of at least one color, gradient of image intensity, andgradient of color intensity.
 11. The camera system of claim 1, whereinthe pre-processing circuit is adapted to perform a weighted histogram bysumming instances of a key over the cells of the grid, wherein the keyincludes the weights as a multiplicative factor, wherein the key is afunction of at least one image parameter selected from the groupconsisting of image intensity, color intensity of at least one color,gradient of image intensity and gradient of color intensity.
 12. Thecamera system of claim 1, wherein the image processor estimatesego-motion of the image sensor, wherein the image processor responsiveto the estimated ego-motion of the image sensor resets at least one theweights of at least one of the cells.
 13. The camera system of claim 1,wherein the image processor estimates image motion of an object in thefield of view of the image sensor and responsive to the estimated imagemotion resets at least one of the weights of at least one of the cells.14. A computerized method for pre-processing image data of an imageframe in an image sensor, the computerized method comprising:associating weights respectively with cells of a grid, wherein theassociating comprises assigning zero weights to cells of the gridlocated outside a region of interest, and assigning non-zero weights tocells of the grid that are included in the region of interest and tocells of the grid that are only partially included in the region ofinterest; generating a pre-processing result only for cells associatedwith non-zero weights; and modifying one or more camera parameters basedon the pre-processing result; wherein the cells respectively include aplurality of contiguous picture elements of the image sensor, andwherein the image sensor is controlled by the one or more cameraparameters.
 15. The computerized method of claim 14, further comprising:grouping contiguously picture elements of the image sensor into thecells of the grid with a number of cells per distance on the surface ofthe image sensor substantially less than the resolution of the imagesensor.
 16. The computerized method of claim 14, further comprising:mounting the image sensor on a vehicle as a part of a driver assistancesystem, wherein the region of interest is determined to include cells ofnon-zero weight which image at least a portion of a road in theenvironment of the vehicle when traveling.
 17. The computerized methodof claim 14, wherein the generation of the pre-processing result isresponsive to the association of the weights.
 18. The computerizedmethod of claim 15, further comprising: resetting a camera parameter ofthe one or more camera parameters responsive to the pre-processingresult.
 19. The computerized system of claim 18, wherein said cameraparameter is an aperture.
 20. The computerized method of claim 14,further comprising: performing a weighted histogram.
 21. Thecomputerized method of claim 14, wherein the pre-processing circuit isadapted to perform a weighted histogram by summing instances of a keyover the cells of the grid, wherein the key includes the weights as amultiplicative factor, and wherein the key is a function of at least oneimage parameter selected from the group consisting of: image intensity,color intensity of at least one color, gradient of image intensity andgradient of color intensity.
 22. The computerized method of claim 14,further comprising: estimating ego-motion of the image sensor;responsive to the estimated ego-motion, resetting at least one of theweights of at least one of the cells.
 23. The computerized method ofclaim 14, further comprising: estimating image motion of an object beingtracked in the field of view of the image sensor; responsive to theestimated image motion, resetting at least one of the weights of atleast one of the cells.
 24. The computerized method according to claim14 wherein the weights differ from the one or more camera parameters.25. The computerized method according to claim 14, further comprisingadaptively change the weights responsive to changes in the scene beingimaged by the image sensor.
 26. The computerized method of claim 14,wherein the pre-processing circuit is adapted to perform a weightedhistogram by summing instances of a key over the cells of the grid,wherein the key includes the weights as a multiplicative factor, andwherein the key is a function of at least one image parameter selectedfrom the group consisting of: gradient of image intensity and gradientof color intensity.
 27. The computerized method of claim 14, wherein theassigning non-zero weights to the cells of the grid that are onlypartially included in the region of interest is based, at least in part,on areas of the cells within the region of interest.
 28. Thecomputerized method of claim 14, wherein the assigning non-zero weightsto the cells of the grid that are only partially included in the regionof interest comprises scaling down the non-zero weight in proportion toareas of the cells within the region of interest.